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EXECUTIVE  SUMMARY 


The  mission  of  JTC3A  is  to  ensure  interoperability  of  tactical  C3  sys¬ 
tems  employed  in  joint  and  combined  operations  of  the  US  Military.  At  the 
present  time,  there  does  not,  <orist  a  systematic  way  of  assessing  JTC3A’s 
effectiveness  in  fulfilling  this  mission.  This  feasibility  study  is  the  first  phase 
of  an  attempt  to  develop  a  methodology  for  building  an  overall  assessment 
system  (OAS)  which  would  allow  JTC3A  to  evaluate  its  effectiveness,  and 
will  be  helpful  in  answering  questions  such  as  how  well  is  the  Agency  ac¬ 
complishing  its  mission?  How  could  its  performance  be  improved,  given  a 
constant  level  of  resources?  What  would  be  the  effect  of  a  change  of  resources 
allocated  to  the  Agency? 


To  determine  the  feasibility  of  developing  such  a  methodology  and 
building  the  suitable  OAS  a  study  of  the  responsibilities,  activities  and  modes 
of  operation  of  the  Agency  was  performed.  Special  attention  was  given  to 
inquiring  into  the  methods  used  by  the  agency  in 


(i)  determining  interoperability  requirements  and  assessing  their  importance; 

(ii)  making  plans  and  schedules  and  establishing  priorities  in  addressing 

these  requirements; 

(iii)  allocating  and  controlling  resources  expended  in  the  process  of  putting 
in  place  the  systems,  procedures  and  devices  needed  for  satisfying  these 
requirements; 

(iv)  assessing  the  effectiveness  of  the  systems,  procedures  and  devices  put 
in  place. 


This  study  led  us  to  the  following  conclusions: 


1.  A  top-down  automated  decision  support  system  approach  appears  to  be 
the  most  suitable  for  building  the  OAS  needed  by  the  Agency. 
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2.  Such  an  OAS  would  require  the  building  of  a  needs  map  database,  specify¬ 

ing  overall  needs  and  how  they  are  satisfied  both  from  the  user  and  the 
Agency’s  viewpoint,  and  a  projects  database ,  specifying  the  projects  of 
JTC3A  and  their  status.  Forerunners  of  such  databases  exist  at  JTC3A 
(The  JIMS  and  Instant  Architecture  Data  Base). 

3.  From  these  databases  several  meaningful  aggregated  and  disaggregated 

measures  of  effectiveness  cam  be  computed  on  a  continuous  basis. 

4.  We  are  confident  that  JTC3A  possesses  the  necessaxy  tadent  to  develop 

such  a  system  wholly  in-house.  The  effort  would  be  commensurate  to 
that  applied  to  the  development  of  other  JTC3A  databases  and  the 
decision  to  do  the  work  wholly  in-  house  or  to  contract  out  parts  of  it 
depends  on  the  priority  given  to  this  project. 

5.  If  the  decision  is  made  to  construct  such  an  OAS  the  next  steps  would  be 

(a)  An  in  depth  study  for  determining  the  measures  (indices)  to  be 
produced  by  the  OAS. 

(b)  Development  of  criteria  and  procedures  for  ranking  importance 
of  interoperability  links  amd  for  determining  users  requirements 
satisfaction. 

(c)  Obtaining  quantitative  estimates  of  amount  of  data  required  and 
ensuing  costs. 

(d)  Design  of  the  OAS. 

(e)  Development  of  the  databases  and  computer  programs. 

(f)  Testing  and  putting  the  OAS  in  place. 


These  steps  should  not  delay  the  continuation  of  the  development  of 
JIMS  and  IADB. 
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INTRODUCTION 


The  objective  of  this  study  is  to  determine  the  feasibility  of  using  exist¬ 
ing  methodologies  and/or  developing  new  ones,  for  assessing  the  effectiveness 
of  JTC3A  in  attaining  its  mission  of  ensuring  the  interoperability  of  tactical 
C3  systems  employed  in  joint  and  combined  operations  of  the  US  Military. 
If  feasible,  an  “assessment  system”  would  allow  JTC3A  to  evaluate  its  effec¬ 
tiveness  in  attaining  its  objectives.  It  will  be  helpful  in  answering  questions 
such  as  how  well  is  the  Agency  accomplishing  its  mission;  how  could  its  per¬ 
formance  be  improved,  given  a  constant  level  of  resources,  and  what  would 
be  the  effect  of  a  change  of  resources  allocated  to  the  Agency. 

Section  1  of  the  study  lists  the  main  capabilities  required  from  JTC3A 
in  accomplishing  its  mission.  The  actual  way  in  which  each  of  these  functions 
is  now  addressed  is  analyzed  in  Section  2.  Goals  of  the  study  are  detailed 
in  Section  3  and  this  leads  to  a  list  of  questions  to  be  answered  concern¬ 
ing  JTC3A’s  objective,  the  measurement  of  how  well  it  is  attained,  required 
data,  possible  methodologies  and  building  of  an  assessment  system.  Section  4 
addresses  the  choice  of  a  methodology  by  considering  two  main  approaches: 
“bottom-up”  and  “top-down”  analysis,  and  then  examining  which  Opera¬ 
tions  Research  and  Management  Science  techniques  appear  to  be  the  most 
likely  to  satisfy  the  Agency’s  needs,  (and  which  ones  do  not  appear  to  be 
promising).  Based  on  this  analysis,  an  outline  of  an  overall  interoperability 
and  effectiveness  assessment  system  is  presented  in  Section  5.  This  assess¬ 
ment  system,  based  on  a  top-down  approach  to  assessment  of  interoperability 
measures,  requires  three  main  components:  (a)  a  “map”  of  interoperability 
needs  of  the  U.S.  Military  (in  the  form  of  an  automated  database),  (b)  a 
database  of  all  JTC3A  projects  and  (c)  a  collection  of  procedures  for  using 
(a)  and  (b)  to  obtain  the  desired  measures.  Possible  effectiveness  and  inter¬ 
operability  measures  are  further  discussed  in  Section  6  together  with  their 
feasible  uses.  Conclusions  regarding  the  feasibility  of  such  an  interoperability 
assessment  system  and  its  development  are  drawn  in  Section  7.  A  list  of  the 
next  steps  to  be  taken  should  the  decision  be  made  to  develop  such  a  system 
completes  the  study. 
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1.  OBJECTIVE  OF  JTC3A 


The  overall  objective  of  the  Agency  is  to  ensure  the  highest  possible  level 
of  interoperability  of  tactical  C3  systems  employed  in  joint  and  combined 
operations  of  the  US  Military.1 

The  attainment  of  this  objective  requires  the  Agency  to  have  the  ca¬ 
pability  to: 


(a)  Determine  present  and  future  needs  of  C3  interoperability  of  the  overall 
Military  system; 

(*>)  Assess  the  importance,  urgency  and  feasibility  of  satisfying  each  of  the 
needs; 

(c)  Prioritize  and  devise  plans  and  schedules  for  addressing  these  needs; 

(d)  Contro.  and  allocate  the  resources  for  developing  the  means  required 

in  the  process  of  putting  in  place  the  systems,  procedures  or  devices 
necessary  for  satisfying  these  needs; 

(e)  Scrutinize  and  cssess  the  degree  of  satisfaction  of  needs  achieved  by  the 

systems,  procedures  or  devices  sifter  they  are  put  in  place; 

(f )  Evaluate  its  own  effectiveness  and  its  impact  on  overall  interoperability. 


These  required  capabilities  are  discussed  in  detail  in  Section  2. 

1In  this  report  C3  includes  intelligence  unless  otherwise  specified.  C3I  will  not  be  used. 
Also  interoperability  will  be  used  to  mean  interoperability  of  tactical  C3  systems  employed 
in  joint  and  combined  operations. 
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2.  A  SHORT  OVERVIEW  OF  THE  PRESENT  STATE  OF  JTC3A. 


Recently  the  Agency  went  through  a  rather  comprehensive  organizational 
restructure,  and  it  is  still  in  a  transitional  mode,  in  the  sense  that  it  did  not 
reach  a  new  steady  state  free  of  the  impacts  due  to  the  organizational  changes. 
Therefore  the  short  overview  provided  here  may  not  represent  accurately  the 
steady  state  mode  of  operation  that  may  be  reached  in  the  not  too  far  ahead 
future.  The  following  overview  follows  the  categories  of  Section  1. 


(a)  The  determination  of  interoperability  needs  comes  from  several  sources: 

(al)  development  of  CINC  architectures  which  entail  a  systematic 
study  of  requirements.  The  product  is  a  detailed  description  and 
assessment  of  C3  interoperability  needs  at  all  echelons  of  the  op¬ 
erations  of  the  CINC; 

(a2)  development  of  functional  architectures  addressing  nine  tactical 
functional  areas  each  divided  into  up  to  16  sub-areas.  These  are 
more  generic  in  nature  than  CINC  architectures  and  the  products, 
which  are  intended  to  be  used  as  building  blocks  in  tailoring  CINC 
and  other  types  of  C3  architectures,  include  descriptions  and  as¬ 
sessments  of  C3  interoperability  needs  across  a  very  wide  range  of 
operations; 

(a3)  interoperability  assessments  of  CINCs  master  plans.  Reviews  of 
CINCs  contingency  and  operations  plans,  service  requirements 
documents,  equipment  and  emerging  technologies  and  represen¬ 
tation  in  standing  boards  and  panels  dealing  with  joint  and  com¬ 
bined  operations; 

(a4)  aggressive  assignment  of  AOs  to  participate  in  meetings  and  dis¬ 
cussions  of  issues  that  may  involve  interoperability  aspects; 

(a5)  requests  from  outside  the  Agency,  that  may  take  in  some  cases 
the  form  of  tasking. 


It  is  worthwhile  mentioning  that  the  automated  Joint/Combined  Issue 
Management  System  (JIMS)  which  has  recently  been  put  into  operation  is 
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a  useful  aid  in  reporting  identification  of  new  needs,  among  other  important 
types  of  services  that  it  provides.  Other  automated  systems  that  may  aid 
in  the  process  of  identifying  needs  are  the  Instant  Architecture  Data  Base, 
IADB,  (needlines  database)  being  developed  by  ARM  in  Reston,  the  ROC- 
requirements  database  and  the  CINC  IPS  database. 


(b  &  c)  Formally,  assessment  of  importance,  urgency  and  feasibility,  and 
prioritizing,  is  well  taken  care  of  by  two  well  defined  processes.  The 
first  is  a  JCS  program  for  annual  review,  coordination  and  resolution  of 
critical  C3  interoperability  requirements  and  issues  (Tactical  C3  Inter¬ 
operability  Improvement  Program  -  IIP)  performed  by  a  General/Flag 
Officer  Prioritization  Board  and  an  0-6  level  Screening  Board.  All 
services,  CINCs  and  Agencies  take  part  in  this  program  and  are  repre¬ 
sented  on  these  boards.  The  Director,  JTC3A  is  the  Executive  Agent 
for  this  program.  The  goals,  responsibilities  and  procedures  are  de¬ 
scribed  in  JTC3A  circular  3101.  In  summary  it  is  clear  that  the  intent 
of  the  program  is  to  enable  identification  of  needs,  to  achieve  coor- 
dir^ion  in  funding,  fielding  and  all  other  aspects,  and  to  agree  on 
priorities. 

The  second  formal  process  is  the  Agency’s  Issue  Resolution  Process 
(IRP)  as  described  in  JTC3A  Instruction  3102  of  March  1990.  This 
process  is  controlled  by  the  Issue  Resolution  Board  (IRB)  and  Screen¬ 
ing  Panel.  In  summary  the  IRB  is  responsible  for  assessing  all  C3  inter¬ 
operability  issues/deficiencies  identified  by  the  Agency  and  for  briefing 
recommended  courses  of  action  to  the  Director  and  Associate  Direc¬ 
tor  for  their  approval.  The  Issue  Resolution  Process  is  supposed  to  be 
aided  by  JIMS  in  identifying  new  issues/deficiencies  and  in  monitoring 
changes  in  status  of  old  ones.  In  effect  the  IRB  is  supposed  to  “deter¬ 
mine  the  priority  of  resolving/taking  action  on  each  issue.  This  is  an 
important  step  as  the  IRB  must  recommend  to  the  Director,  JTC3A 
whether  resources  need  to  be  reprogrammed  to  address  an  issue  as  well 
as  to  establish  a  prioritized  listing  of  all  issues  which  need  resources  in 
order  to  be  fully  addressed”.  (JTC3A  Instructions  3102). 

Our  impression  concerning  (b  &  c),  formed  by  interviews  with  Agency 
personnel,  is  that  evaluation  of  importance  and  urgency  of  the  needs 
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is  mostly  decentralized.  Ranking  of  the  issnes  to  be  addressed  is  done 
mostly  at  the  level  of  the  directorates,  where  the  Deputy  Director  is  re¬ 
sponsible  for  preparing  the  ranking  list  and  matching  it  with  the  direc¬ 
torate  resources  in  order  to  decide  where  to  put  the  cutting  point.  The 
ranked  lists  are  scrutinized  by  the  Director  to  ensure  reasonable  rank¬ 
ing  and  then  brought  before  the  Resource  Allocation  Committee  (RAC) 
where  some  fine  tuning  adjustments  are  decided  upon.  RAC  is  a  top 
echelon  committee  (Director,  Chief  of  Staff,  Associate/Deputy/ Assistant 
Directors). 

The  Agency’s  organizational  structure  is  definitely  designed  to  accom¬ 
modate  the  three  major  activities  namely:  standards,  interoperability 
assurance  (including  architectures)  and  testing/certification.  The  re¬ 
sources  at  the  disposal  of  the  Agency  are  determined  by  the  executive 
and  legislative  branches  of  the  U.  S.  government.  Once  approved,  the 
Agency  seems  to  have  a  rather  high  degree  of  autonomy  in  allocating 
its  resources  and  is  also  not  banned  from  asking  for  matching  funds  or 
even  complete  financial  sponsoring  for  some  specific  requests  coming 
from  without.  However  the  common  case  is  for  a  project  to  be  sup¬ 
ported  from  the  budgeted  resources  of  the  Agency.  One  of  the  three 
major  activities  where  the  flexibility  in  allocation  and  deployment  of 
resources  is  more  limited  is  the  testing/certification  activity.  This  is 
due  to  the  requirement  that  all  equipment  must  be  certified  and  thus 
tested.  The  effectiveness  of  this  activity  should  essentially  be  measured 
by  its  operational  efficiency  and  quality  level  achieved. 

There  does  not  seem  to  exist  at  present  a  single  organ  responsible  for 
this  feedback  required  capability  in  the  Agency.  This  is  not  to  say  that 
most  or  even  all  the  needed  information  is  not  within  reasonably  easy 
reach  of  the  Agency.  AOs  are  in  most  cases  aware  of  what  actually 
was  implemented,  and  to  what  degree,  as  far  as  their  own  projects 
are  concerned.  We  were  unable  to  identify  an  in-place  commonly  used 
procedure  for  reporting  this  information  in  a  standardized  form  to  a 
centred  data  bank,  automated  or  otherwise. 

The  Agency  is  involved  in  a  relatively  large  number  of  projects  at  any 
point  in  time.  At  the  present  time  the  Agency  does  not  have  at  its  dis¬ 
posal  tools  for  evaluation  of  its  overall  performance  in  terms  of  overall 
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effectiveness  and  contribution  to  interoperability  of  the  U  S.  Military. 
This  is  not  to  say  that  conventional  procedures,  such  as  Annual  Reports 
and  the  like  are  not  in  place. 

One  step  towards  an  overall  self-assessment  system  has  been  already 
taken  in  the  form  of  JIMS  which  is  an  automated  system  enabling 
the  Agency  to  obtain  summary  information  about  all  issues  treated 
by  the  Agency.  This  system  is  accessible  to  practically  all  levels  of 
the  organization  and  is  updated  on  a  continuous  basis.  It  allows  to 
easily  identify  the  issues  needing  more  urgent  attention  and  it  has  the 
capability  to  provide  quantitative  summary  information  in  the  form  of 
graphs  and  numbers.  It  is  also  a  useful  tool  that  can  be  used  on  a 
day-to-day  basis  by  all  levels  of  the  organization  in  identifying  ongoing 
and  finished  projects  and  issues  related  to  their  own  work.  This  system 
is  quite  new  and  is  still  developing.  It  may  prove  to  be  useful  as  one 
element  in  an  overall  assessment  system,  if  such  a  system  is  to  be  built. 
A  second  promising  system  is  the  Instant  Architecture  Data  Base  being 
developed  by  ARM  in  Reston.  This  system  when  completed  is  designed 
to  store  information  on  needlines,  to  identify  feasible  equipment  strings, 
check  for  certification  and  be  used  as  an  automated  tool  in  Architecture 
development.  When  successfully  completed  it  may  also  serve  as  a  major 
building  block  in  an  overall  assessment  system.  Two  other  automated 
systems  that  may  prove  to  be  useful  for  that  purpose  are  the  ROC  and 
CINC  IPS  databases. 


3.  GOALS  OF  THIS  STUDY 


Although  JTC3A  commits  considerable  resources  towards  improving  the  level 
of  C3  interoperability,  no  systematic  way  of  assessing  its  effectiveness  exists 
at  the  present  time.  This  “feasibility  study”  is  the  first  phase  of  an  attempt 
to  develop  such  a  methodology.  An  “assessment  system”  would  allow  JTC3A 
to  evaluate  its  effectiveness  in  attaining  its  objective.  It  will  be  helpful  in  an¬ 
swering  questions  such  as  how  well  is  the  Agency  accomplishing  its  mission? 
How  could  its  performance  be  improved,  given  a  constant  level  of  resources? 
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What  would  be  the  effect  of  a  change  of  resources  allocated  to  the  Agency? 
A  possible  second  phase  would  be  the  development  and  implementation  of 
the  methodology  identified  in  the  first  phase,  provided  such  a  methodology 
exists  or  may  be  developed. 

In  order  to  answer  the  above  questions  this  study  considers: 

1)  The  objective  of  JTC3A  and  how  and  to  what  degree  it  may  be  defined 

in  an  operational  way. 

2)  How  can  the  extent  to  which  the  objective  (or  a  subobjective)  has  been 

achieved  be  measured  or  otherwise  determined. 

3)  What  data  are  available  regarding  the  JTC3A  activities  with  direct  im¬ 

pact  on  its  main  objective  in  the  short  run,  and  what  data  are  available 
which  influence  its  main  objective  in  the  medium  or  long  run. 

4)  What  additional  data  should  be  obtained,  and  from  what  sources. 

5)  What  existing  methodologies  could  be  used  in  an  overall  assessment  sys¬ 

tem.  Should  a  new  methodology  be  developed?  Which  methodology 
appears  to  be  the  most  promising  one? 

6)  What  are  the  main  steps  in  the  development  of  an  overall  assessment 

system  implementing  that  methodology. 

7)  Could  such  a  system  be  developed  by  JTC3A. 


4.  CHOICE  OF  A  METHODOLOGY 


JTC3A  has  a  single,  well-defined  major  mission,  i.e.  to  improve  the  level  of 
interoperability  of  US  Military  tactical  C3  systems  used  in  joint  and  com¬ 
bined  operations.  However,  the  JTC3A  activities  which  contribute  to  the 
achievement  of  this  goal  are  numerous  and  diverse.  Obtaining  an  operational 
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definition  of  level  of  interoperability  and  estimating  this  level  is  a  complex 
bask  involving  large  amounts  of  data.  Moreover,  the  time  dimension  appears 
to  be  a  crucial  one  as  a  number  of  Agency  activities,  rather  than  addressing 
present  C3  interoperability  needs,  are  geared  at  improving  future  interop¬ 
erability  effectiveness,  when  technology  now  planned  or  in  the  development 
phase  will  become  operationally  available. 

One  possible  approach  to  assessing  JTC3A  “effectiveness”  would  be  a 
“bottom-up”  one:  Agency  activities  would  first  be  partitioned  with  a  suf¬ 
ficient  amount  of  detail  into  homogeneous  categories  (e.g.  designing  func¬ 
tional  or  CINC  architectures,  planning  communications  for  anti-drug  joint 
and  combined  operations,  etc.)  for  each  of  which  a  subobjective  could  be 
defined.  Then  for  each  of  these  categories  of  activities  an  assessment  method 
would  be  developed  to  estimate  the  level  of  attainment  of  the  subobjective 
(e.g.  number  of  Mil  standards  developed,  percentage  of  times  a  commer¬ 
cial  standard  could  be  used).  Finally,  the  degree  of  attainment  estimates 
for  the  various  activity  categories  would  be  aggregated  into  an  estimate  of 
global  degree  of  attainment.  Such  an  approach  has  some  advantages.  The 
efficiency  of  some  activities  of  JTC3A  can  be  estimated  using  standard  meth¬ 
ods  (e.g.  industrial  engineering  techniques  for  the  Test  Center).  Criteria  for 
other  activities  which  reflect  a  major  Agency  and/or  DoD  concern  might 
be  adopted  (e.g.  the  degree  to  which  commercial  standards  can  be  used 
instead  of  Agency  ones).  However,  there  are  also  some  considerable  difficul¬ 
ties.  Namely  it  would  be  hard  to  answer  the  two  following  questions:  (i)  how 
should  estimates  of  attainments  and  efficiency  levels  for  the  various  activities 
be  translated  in  a  meaningful  way  into  an  overall  measure  of  effectiveness, 
(ii)  how  could  these  estimates  be  used  to  find  how  to  increase  effectiveness 
with  the  given  amount  of  resources  or  to  estimate  what  would  be  the  effect 
of  a  change  in  level  of  resources  on  effectiveness?  The  bottom-up  approach 
focuses  on  the  question  of  measuring  “how  well  the  Agency  is  doing  what  it  is 
doing”.  This  is  an  introspective  approach  which,  if  successfully  implemented, 
will  measure  efficiency  rather  than  effectiveness,  where  efficiency  is  defined 
as  effectivity  of  operational  performance  and  effectiveness  is  defined  in  terms 
of  impact  on  the  interoperability  of  the  U.  S.  military  system. 

Another  approach  to  assessing  JTC3A  effectiveness  would  be  a  “top-- 
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down”  one:  the  need  for  interoperability,  would  first  be  established  in  suffi¬ 
cient  detail  both  for  the  present  and  for  the  near  future  (i.e.  impact  of  new 
needlines,  of  new  technologies  both  on  the  US  and  Allies  and  on  the  oppo¬ 
nents  side  would  be  evaluated);  then  measurements  would  be  made  (following 
criteria  to  be  discussed  below)  of  the  degree  to  which  the  actual  interoper¬ 
ability  needs  are  satisfied  and  estimates  devised  for  the  value  of  this  measure 
in  the  near  future.  The  evolution  of  these  figures  in  time  would  constitute  an 
aggregate  measure  of  Agency  effectiveness.  Then  the  impact  of  the  various 
Agency  activities  on  unsatisfied  present  needs  and  on  likely  to  be  unsatisfied 
future  needs  would  be  studied.  This  would  give  a  tool  to  improve  effectiveness 
with  the  present  level  of  resources  by  changing  priorities  of  various  projects 
of  the  Agency,  as  well  as  to  evaluate  the  impact  on  effectiveness  of  changes 
in  level  of  resources.  The  top-down  approach  focuses  on  the  interoperability 
needs  of  the  military  system  and  attempts  to  measure  the  impact  of  what 
the  Agency  is  doing  on  the  overall  “interoperability  level”.  It  is  effectiveness, 
rather  than  efficiency,  oriented. 

Such  an  approach,  if  possible  to  pursue  and  implement,  would  answer 
the  main  questions  discussed  in  Section  3.  Its  feasibility  is  analyzed  in  more 
detail  in  the  next  Sections. 


Assuming  a  top-down  approach  to  effectiveness  assessment  is  chosen, 
the  next  question  is  how  formal  should  such  an  approach  be?  Is  it  reasonable 
to  use  mathematical  modelling  to  a  large  extent?  Could  some  existing  “hard” 
O.R.  methodologies  be  brought  to  bear  or  should  the  effectiveness  assessment 
system  be  closer  to  an  interactive  Decision  Support  System ? 

In  the  remainder  of  this  Section,  conclusions  regarding  O.R.  method¬ 
ologies  are  stated  and  briefly  explained. 

Mathematical  Programming  (Linear,  nonlinear  and  stochastic)  mod¬ 
els  have  been  used  extensively  to  solve  resource  allocation  problems.  Such 
models  are  useful  when  (i)  sufficiently  precise  and  measurable  data  on  the  ef¬ 
fectiveness  of  activities  and  the  unit  amounts  of  resources  they  use  are  known 
(possibly  in  the  form  of  probability  distributions)  including  their  change  pat- . 
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tern  in  time;  (ii)  when  the  levels  of  activities  may  be  fractional  and  (iii)  the 
objective  function  expressing  effectiveness,  and  constraints  expressing  use  of 
resources  have  additional  properties  making  the  problem  solvable  by  current 
algorithms  (e.g.  they  are  linear  in  each  variable  in  linear  programming).  The 
very  large  amount  of  data  required,  its  rapid  changing  in  time,  the  fact  that 
many  activities  must  be  done  entirely  or  not  at  all,  and  that  the  objective 
function  expressing  effectiveness  could  have  a  difficult  to  handle  form  render 
this  approach  unattractive. 

Research  and  Development  projects  selection  models  and  methods  which 
often  use  mathematical  programming  in  0*1  variables  do  not  suffer  from  al¬ 
lowing  fractional  implementation  of  projects,  but  all  other  difficulties  cited 
above  remain. 


Multiattribute  Utility  Theory  can  in  principle  be  used  to  rank  projects 
judged  along  several  dimensions.  It  does  not  seem  to  apply  well  to  complex 
constrained  problems  such  as  the  one  considered  here.  Moreover,  when  there 
are  several  relevant  dimensions  eliciting  the  utility  function  it  is  very  difficult 
and  sometimes  arbitrary  in  practice. 


Methods  of  Multiple  Criteria  Decision  Making  (e.g.  the  Analytical  Hi¬ 
erarchy  Process  and  Electre  Methods),  also  consider  discrete  sets  of  projects 
to  be  ranked  or  categorized.  While  none  of  these  does  seem  comprehensive 
enough  to  accommodate  the  JTC3A  effectiveness  problem,  due  to  its  size 
and  the  interactions  between  individual  projects,  they  could  be  useful  as  de¬ 
cision  aid  tools  for  solution  of  important  subproblems  with  a  strong  impact 
on  JTC3A  effectiveness.  One  such  instance  would  be  project  selection  in  the 
RAC. 


In  conclusion,  it  does  not  appear  that  any  existing  O.R.  hard  method¬ 
ology  is  adequate  for  expressing  the  overall  problem  of  maximizing  JTC3A  ef¬ 
fectiveness,  and  simultaneously  estimating  its  present  level  and  how  it  can  be 
improved  by  reallocation  of  resources.  The  decision  support  system  method¬ 
ology,  in  which  information  stored  in  a  database,  or  in  linked  databases,  sum¬ 
marized  by  various  statistical  techniques  and  constantly  updated,  is  made 
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available  to  decision  makers,  seems  to  be  much  more  promising. 


5.  OUTLINE  OF  AN  OVERALL  INTEROPERABILITY  AND 
JTC3A  EFFECTIVENESS  ASSESSMENT  SYSTEM 


The  purpose  of  the  overall  assessment  system  (OAS)  is  to  provide  aggre¬ 
gated  measures  of  the  interoperability  state  and  the  Agency’s  effectiveness 
within  a  reasonably  short  time,  whenever  required.  The  level  of  aggregation 
determines  the  dimensionality  of  the  measures.  Ideally,  at  the  uppermost 
aggregation  level  these  measures  are  uni-dimensional,  namely,  overall  inter¬ 
operability  is  measured  by  one  number  and  overall  effectiveness  is  measured 
by  one  number.  At  lower  levels  of  aggregation  the  measures  may  take  the 
form  of  vectors  or  matrices  (tables)  of  numbers.  The  Agency  may  desire 
an  OAS  with  a  capability  of  providing  assessment  at  more  than  one  level  of 
aggregation. 

A  uni-dimensional  measure  enables  sharp  determination  of  improve¬ 
ment  or  deterioration  in  the  measured  organization  and  a  simple  plotting  of 
its  evolvement  in  time  enables  to  determine  in  a  glance  positive  or  negative 
trends.  The  deficiency  of  such  a  measure  is  its  fuzzy  meaning.  A  unidi¬ 
mensional  measure  is  actually  an  index  composed  of  several  factors  (uni¬ 
dimensional  lower  level  measures)  each  measured  in  different  units.  The  ag¬ 
gregation  of  these  factors  into  one  index  involves  subjective  elements  or,  at 
best,  a  rather  crude  representation  of  their  real  weights.  A  multi-dimensional 
measure  is  more  precise  and  meaningful  since  each  factor  is  represented  in¬ 
dividually.  Such  a  measure  contains  more  accurate  information  but  may  be 
less  sharp  in  determination  of  overall  change  and  trends. 


The  OAS  outlined  here  is  based  on  the  top-down  approach  to  assess¬ 
ment  of  the  measures  of  interest.  For  such  an  OAS  to  be  operative  it  is 
required  that  the  three  following  major  components  be  in  place. 
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(a)  An  updated  “map”  of  interoperability  needs  of  the  U.S.  military  (in  the 

form  of  an  automated  database). 

(b)  An  easily  accessible  database  of  all  completed,  ongoing  and  scheduled 

JTC3A  projects. 

(c)  A  collection  of  procedures  (software)  for  using  (a)  and  (b)  to  produce 

the  desired  assessment  measures  at  various  aggregation  levels. 


(a)  Needs 


The  overall  military  machine  can  be  viewed  as  a  large  collection  of  commu¬ 
nicating  entities  (nodes),  some  of  which  exist  on  an  ongoing  basis  and  some 
planned  to  come  into  existence  if  and  when  some  foreseeable  or  unforesee¬ 
able  situations  develop.  An  interoperability  needline  (or  link)  is  a  pair  of  such 
nodes,  not  belonging  to  the  same  service,  agency  or  country,  which  have  a 
communications  connectivity  need  for  exchanging  information  for  performing 
their  functions  on  a  permanent  basis,  or  in  case  that  some  specific  conditions 
develop.  The  needs  graph  is  defined  as  the  set  of  all  nodes  and  their  interop¬ 
erability  needlines  (links  or  arcs).  When  complete  information  is  attached  to 
each  link  this  network  becomes  the  map  of  interoperability  needs.  It  should 
be  noted  that  although  the  needs  graph  for  the  U.S.  is  quite  large,  the  needs 
map  to  be  maintained  in  an  automated  database  is  not  prohibitively  large 
since  the  majority  of  needlines  are  generic  and  may  exist  in  all  or  most  CINCs 
in  identical  form. 


The  essential  information  required  for  each  needline  is: 


1.  Type  of  functions  for  which  information  exchange  is  required. 

2.  Preferred  and  actual  modes  of  communications,  such  as  voice,  data,  tele¬ 

type,  facsimile,  video  or  any  combination  therein.  Information  on  spe¬ 
cific  parameters  should  also  be  specified  when  relevant.  Modes  which 
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are  critical  should  be  so  marked.  Other  modes  desirability  should  be 
ranked. 


3.  Generic  or  CINC/ Agency /Taskforce  specific  classification,  (generic  or 

specific  identification  code). 

4.  Security  and  ECCM  requirements. 

5.  Existing  and/or  feasible  equipment  strings. 

6.  Existing  and/or  possible  message  format,  protocols  and  operating  proce¬ 

dures. 

7.  Existence  of  standards  and  specifications. 

8.  Existence  of  certification. 

9.  Dependence  on  other  links.  A  link  may  be  operative  but  still  of  little 

contribution  to  interoperability  if  its  usefulness  is  conditional  on  a  set 
of  one  or  more  other  links  being  operative.  Full  interoperability  in  such 
a  case  is  achieved  when  all  links  of  the  interdependent  set  are  operative. 
The  interdependent  set  must  be  specified. 

10.  Importance  of  the  link.  This  can  be  of  ordinal  form,  e.g.  a  categori¬ 

cal  rank  on  a  predetermined  set,  starting  from  desired  and  ending  at 
critical.  The  importance  of  the  link  must  reflect  mainly  the  expected 
potential  overall  damage/loss  due  to  its  non-  operability. 

11.  Present  satisfaction  of  interoperability  requirements  from  the  user’s  view¬ 

point  (user’s  operability  number  UON).  This  is  recommended  to  be  a 
simple  representation  of  the  present  state.  Either  (1  -  nonoperative,  0  - 
operative)  or  (1  -  nonoperative,  1/2  -  partially  operative,  0  -  operative). 

12.  Future  UON.  A  forecasted  UON  over  the  predetermined  future  time 

horizon  of  the  OAS. 


It  should  be  noted  that  most  of  the  needlines  information  on  the  map 
must  relate  to  the  present  status  and  to  the  forecasted  status  over  a  pre¬ 
determined  planning  time  horizon.  Forecasts  must  bring  into  consideration 
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technological  developments  and  innovations  by  the  U.S.  and  its  allies  and 
by  its  potential  foes  as  well  as  probable  political  changes  and  developments 
affecting  the  missions,  organizations  and  priorities  of  the  U.S.  military. 

The  needs  map  database  can  serve  to  assess  a  variety  of  measures 
of  overall  interoperability  and  as  an  aid  in  the  assessment  of  the  Agency’s 
effectiveness  and  the  potential  contribution  of  added  resources. 


A  simplistic  example  is  a  users  overall  deficiency  (non-  interoperabil¬ 
ity)  index  (UODI).  By  multiplying  each  needline  UON  by  its  importance 
rank  (assuming  importance  is  ranked  numerically)  and  summing  over  all  the 
needlines  (and  correcting  for  interdependence)  we  obtain  a  uni-dimensional 
measure  of  deficiency.  If  there  are  no  interoperability  deficiencies  this  mea¬ 
sure  (index)  will  take  the  value  zero,  and  will  increase  in  value  as  the  inter¬ 
operability  level  deteriorates.  At  each  point  in  time  it  is  possible  to  have  a 
curve  of  overall  deficiency  measure  starting  at  that  point  and  ending  at  the 
OAS  time  horizon.  As  mentioned  earlier  such  an  index  is  of  the  uppermost 
aggregation  level  with  the  advantages  and  disadvantages  as  mentioned. 


(b)  JTC3A  Projects  Data  Base 

The  essential  information  required  for  each  project  is 

1.  Description,  classification  and  identification  of  the  project. 

2.  Needlines  affected. 

3.  Starting  time  and  expected  successful  completion  time. 

4.  Resources  used  and  expected  resources  expenditure  to  completion. 

5.  Estimated  probability  of  time  and  degree  of  implementation  when  com' 

pleted. 
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6.  Status  of  completed  projects  (e.g.  abandoned,  completed  but  not  im¬ 

plemented,  completed  and  partially  implemented,  completed  and  fully 
implemented,  completed  and  expected  to  be  implemented  at  some  spec¬ 
ified  future  time,  etc.) 

7.  Other  information  not  required  by  the  OAS  but  needed  for  the  other  uses 

of  this  database. 


Here  as  in  the  needs  map  database,  the  information  should  relate  to 
the  present  time  and  to  future  expectations,  whenever  relevant.  From  the 
viewpoint  of  the  OAS  structure  there  is  no  need  to  separate  the  needs  map 
database  from  the  projects  database.  However  the  later  may  serve  as  a 
very  effective  tool  in  monitoring  and  controlling  the  ongoing  activities  of  the 
Agency  at  both  the  micro  and  the  macro  levels  and  should  possess  a  stand 
alone  capability. 

The  projects  database,  in  combination  with  the  needs  map  database, 
enables  the  Agency  to  assess  the  overall  military  system  and  the  Agency’s  roll 
from  more  than  one  viewpoint.  While  as  the  users  index  of  non-interoperability 
is  valid  and  very  meaningful  from  the  users  point  of  view,  it  does  not  provide 
the  complete  interoperability  picture.  A  military  operation  launched  at  a 
specific  point  in  time,  whether  planned  or  unexpected,  will  have  to  make  do 
with  all  the  deficient  needlines  at  that  time.  This  may  result  in  increased 
costs,  in  casualties  and  material,  in  time  delays,  or  in  impairment  of  the 
operation.  The  user  must  thus  be  concerned  with  the  present  deficiencies  of 
the  needlines  involved.  The  information,  for  example,  that  a  certain  major 
deficiency  is  about  to  be  rectified  in  three  months  time  is  an  important  part 
of  the  interoperability  picture  but  is  much  less  relevant  to  the  user  whose 
needs  are  at  present. 

A  complementary  index  is  an  Agency  overall  deficiency  index  (AODI). 
Here  we  may  use  for  each  needline  an  Agency  operability  number  (AON) 
between  0  and  1,  where  AON  =  0  means  100%  operative,  AON  =  1  means 
non-operative  and  no  project  for  rectifying  the  deficiency  is  eith^r  underway 
or  scheduled.  An  AON  number  between  0  and  1  represents  the  percent  of  the 
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remaining  process  for  eliminating  the  deficiency.  This  will  be  calculated  from 
the  data  of  (b3),  (b4)  and  (b5).  The  AODI  is  computed  in  the  same  way  as 
the  UODI  using  the  AONs  instead  of  the  UONs.  Since  for  each  needline  the 
AON  <  UON  then  AODI  <  UODI.  The  AODI  is  a  measure  of  the  overall 
deficiency  which  brings  into  consideration  the  resources  and  time  expended 
by  the  Agency  in  rectifying  needlines  deficiencies  weighted  by  the  importance 
of  the  needlines.  These  weighted  expended  resources  equal  in  fact  to  UODI 
minus  AODI.  (Note  that  the  above  described  AODI  is  suitable  for  the  case 
where  the  UON  can  take  only  the  values  0  or  1.  If  the  UON  is  also  allowed 
to  take  the  value  1/2,  for  partially  operative,  the  value  of  the  AON,  must  be 
normalized  multiplying  it  by  the  UON  value). 

The  UODI  and  AODI  are  brought  here  mainly  as  an  indication  of 
possible  types  of  measures.  We  shall  not  further  analyze  their  properties  at 
this  stage.  An  outline  of  possible  types  of  producible  measures  and  their  uses 
is  presented  in  Section  6. 


(c)  Procedures 


The  purpose  of  the  procedures,  in  the  form  of  standard  software  packages 
(e.g.  DBASE)  and  specially  tailored  computer  programs,  is  to  enable  to 
maintain  and  update  the  databases,  to  access  the  necessary  information  for 
the  construction  of  the  various  aggregated  measures,  to  produce  these  mea¬ 
sures  and  present  them  in  the  desirable  formats  and  to  create  a  user  friendly 
flexible  system  which  can  be  expanded  to  provide  useful  byproducts  upon 
request. 
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6.  OUTLINE  OF  POSSIBLE  EFFECTIVENESS  AND  INTER¬ 
OPERABILITY  MEASURES 


In  this  Section  we  present  outlines  of  several  possible  measures.  The  purpose 
is  not  to  provide  an  exhaustive  description  of  all,  or  even  of  most,  possible 
useful  measures.  Rather,  we  try  to  provide  a  sense  of  types  of  measures  which 
can  be  produced  by  the  suggested  OAS. 


(a)  Measures  of  Non-interoperability 


In  the  previous  Section,  we  described  two  uppermost  aggregation  level  mea¬ 
sures  of  deficiency,  namely,  the  UODI  representing  the  user’s  viewpoint  and 
the  AODI  containing  more  information  and  representing  the  hidden  poten¬ 
tial  due  to  Agency’s  effort.  Both  of  these  measures  are  uni- dimensional. 
Multi-  dimensional  measures  (indexes)  are  produced  by  breaking  down  the 
deficiencies  into  several  categories,  each  of  particular  interest.  One  example 
is  breakdown  by  main  cause  of  deficiency: 


-  new  needs 

-  outdated  equipment 

-  architecture  (missing  or  incomplete) 

-  standard  (incomplete  or  not  accurate) 

-  testing  and  certification  (shortage  or  inadequacy) 

-  low  priority  (budget  consideration) 

-  disagreement  between  services/agencies/other  organizations 

-  other. 
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Clearly  causes  of  deficiency  in  any  specific  needline  are  not  necessarily  mutu¬ 
ally  exclusive,  as  several  factors  may  be  involved  in  causing  any  one  specific 
deficiency.  The  breakdown  here  is  by  main  cause,  where  secondary  causes 
may  be  mentioned.  Thus  we  divide  our  total  map  into  a  number  of  mutually 
exclusive  and  exhaustive  subsets  each  characterized  by  the  main  cause  of 
deficiency.  It  is  possible  now  to  use  the  UON  numbers  and  the  importance 
ranks  and  produce  a  UODI  for  each  one  of  these  subsets.  These  must  sum 
to  the  total  UODI.  The  percentage  contributed  by  each  cause  is  the  ratio  of 
.UODI  of  this  cause  divided  by  the  total  UODI.  This  will  give  a  relatively  clear 
picture  of  which  are  the  causes  that  contribute  most  to  non-  interoperability. 
AODI  can  be  calculated  in  a  similar  manner. 


As  mentioned  earlier  these  multi-dimensional  measures  are  more  mean¬ 
ingful  in  the  sense  that  each  component  represents  one  factor.  However  a 
change  over  time  in  such  a  measure  may  not  always  be  obviously  categorized 
as  positive  or  as  negative  since  some  components  may  improve  and  some  may 
deteriorate.  It  is  also  possible  to  produce  statistics  by  secondary  causes  given 
the  main  cause.  Such  measures  are  of  a  lower  level  of  aggregation.  Other 
examples  of  possible  breakdowns  are: 


-  breakdown  according  to  CINCs/Task  Forces /Allies; 

-  breakdown  according  to  pairs  of  services  (there  are  four  services  which 

produce  six  possible  pairs  such  as  Navy/ Army,  Navy/Air  Force,  etc.) 


Many  other  possibilities  exist  and  the  Agency  will  have  to  decide  which 
are  of  sufficient  interest  and  usefulness  to  be  calculated. 


(b)  Measures  op  Effectiveness 


On  the  face  of  it,  the  measures  of  non-interoperability  do  not  reflect  the 
Agency’s  contribution  to  the  system.  However,  we  have  already  remarked  on 
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the  fact  that  the  difference  between  the  UODI  and  the  AODI  is  actually  a 
representative  of  the  efforts  invested  by  the  Agency  in  unresolved  deficiencies 
at  any  given  point  in  time. 


There  may  be  several  approaches  to  assessing  and  monitoring  the  over¬ 
all  Agency’s  effectiveness.  One  way  is  by  noting  that  every  time  the  Agency 
completes  a  project  successfully  (successfully  includes  implementation  in  the 
field)  the  gap  between  the  UODI  and  the  AODI  decreases  (the  magnitude 
6f  this  decrease  equals  to  the  UON  number  multiplied  by  the  importance 
rank  and  summed  over  all  the  affected  needlines).  To  measure  the  contri¬ 
bution  of  the  Agency  we  select  a  reference  date,  say  January  1.  For  this 
date  we  calculate  the  UODI  from  the  needs  map  database.  We  then  forecast 
the  UODI  over  a  period  of  time  (say  one  year)  by  using  the  information  in 
the  projects  database  to  estimate  the  expected  time  of  completion  of  ongo¬ 
ing  projects  and  the  probability  of  their  success.  At  the  end  of  this  period 
of  time  we  can  calculate  the  actual  UODI  of  the  reference  needs  map  (ex¬ 
cluding  new  needlines)  and  then  compute  the  difference  between  the  actual 
and  forecasted  (actual  minus  forecasted)  indices.  A  zero  difference  is  inter¬ 
preted  as  100%  effectiveness  in  execution  of  the  ongoing  and  planned  projects 
(this  does  not  indicate,  however,  if  the  allocation  of  resources  in  the  form  of 
projects  selected  for  execution  was  optimal  to  begin  with).  If  the  difference 
is  positive  then  the  effectiveness  is  less  then  100%  and  can  be  numerically 
calculated.  Symmetrically  if  the  difference  is  negative  the  effectiveness  is 
more  than  100%. 


In  the  next  time  period  the  same  process  is  repeated  with  the  reference 
needs  map  including  the  new  needlines  which  were  added  during  the  previous 
time-period.  (Note  that  in  the  case  of  drastic  changes  in  the  needs  map 
or  the  resource  allocation  the  time  period  must  be  terminated  and  a  new 
time  period  started).  Over  several  time  periods  it  is  possible  to  identify 
a  trend  in  the  execution  effectiveness  of  the  Agency.  It  is  also  possible  to 
calculate  the  estimated  contribution  for  each  time  period  by  the  decline  of  the 
actual  UODI.  However,  the  meaning  of  the  units  in  which  this  contribution 
is  measured  is  somewhat  fuzzy. 


22 


As  mentioned,  the  above  procedure  will  not  provide  information  about 
the  effectiveness  of  allocating  resources  in  an  optimal  or  even  only  a  good 
way.  The  resources  should  be  allocated  to  projects  with  highest  expected 
returns,  where  the  return  is  measured  by  the  total  impact  per  dollar  invested 
in  this  project  on  the  overall  interoperability.  Impact  per  dollar  is  calculated 
as  follows:  the  theoretical  impact  is  the  expected  magnitude  of  reduction 
in  the  UODI  divided  by  the  total  budget  allocated  to  the  project  assuming 
that  the  project  is  completed  100%  successfully.  This  theoretical  impact 
must  then  be  averaged  by  the  success  probability  distribution  to  obtain  the 
expected  impact.  All  the  required  information  should  bs  available  since  it  is 
contained  in  the  two  databases.  In  a  near-optimal  allocation  the  projects  are 
ordered  according  to  decreasing  expected  impacts  except  for  the  few  lowest 
priority  projects  near  the  cutting-point,  where  it  may  be  necessary  to  deviate 
from  the  order  for  the  purpose  of  keeping  within  the  available  budget  and 
other  resources. 


If  the  need  arises  to  compare  several  plans  of  resource  allocation  it  is 
possible  to  compute  the  expected  future  impact  of  each  plan  on  the  total 
UODI  or  on  lower  level  of  aggregation  UODIs.  A  comparison  of  the  result¬ 
ing  forecasts  may  help  in  identifying  the  best  plan  or  in  ranking  the  plans. 
This  may  be  of  use  at  the  level  of  strategic  planning  involving  decisions  to 
change  the  allocation  of  resources  between  areas  of  activity  (standards,  ar¬ 
chitectures,  testing  and  certification,  overall  monitoring  of  the  needs  of  the 
military  system,  feedback  flow  enhancement,  etc.). 


Other  measures  of  effectiveness  are  possible  and  the  ones  to  be  used 
are  subject  to  the  Agency’s  preferences. 


(c)  The  Impact  of  Resource  Changes 


The  impact  of  changes  in  resources  can  be  measured  in  terms  of  forecasted 
expected  change  in  the  overall  interoperability  (e.g.  the  UODI)  and/or  in  its 
components.  Increasing  or  decreasing  resources  without  change  in  policies 
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and  priorities  will  essentially  affect  the  lowest  priority  projects.  Frequently 
resource  changes  are  a  result  (or  a  cause)  of  changes  in  DoD  policies  or 
priorities.  In  such  cases  the  needs  map  database  must  be  updated  to  reflect 
the  new  information  and  all  projects  must  be  re-ranked.  The  calculated 
expected  impact  on  the  overall  interoperability  will  reflect  both  resource  and 
policy  changes. 


At  the  close  of  this  section  it  is  worthwhile  pointing  out  that  effective¬ 
ness  does  not  necessarily  imply  efficiency  at  the  micro  level.  The  OAS  is 
a  tool  intended  for  macro  assessment  of  effectiveness  in  use  and  allocation 
of  resources.  The  efficiency  of  performance  at  the  individual  projects  level 
cannot  be  assessed  by  the  OAS.  Here  the  assessment  tools  are  conventional 
efficiency  studies  using  industrial  engineering  methods,  and  there  is  no  es¬ 
cape  from  breaking  down  individual  projects  into  the  collection  of  activities 
and  phases  involved  and  studying  these  for  possible  efficiency  improvements. 


7.  FEASIBILITY 


While  the  proposition  of  developing  a  quantitative  OAS  based  on  “hard” 
Operations  Research  and  Decision  Sciences  methodologies  is  rather  question¬ 
able  we  find  the  decision  support  system  approach  much  more  suitable  for 
the  Agency’s  needs.  Thus  we  rule  out  the  possibility  of  developing  a  highly 
structured  optimization  system  as  being  non-practical  and  concentrate  on 
the  feasibility  of  a  system  with  features  as  outlines  in  Section  5.  For  such  a 
system  the  feasibility  issue  breaks  into  two  questions: 


(1)  Is  it  feasible  to  assemble  all  the  required  information,  organize  it  in  the 

databases,  develop  the  necessary  non-standard  software  and  maintain 
the  system  on  a  permanent  ongoing  basis? 

(2)  Assuming  the  answer  to  (1)  is  positive,  is  it  feasible  to  extract  from  this 

system  meaningful  measures  of  overall  effectiveness  and  interoperabil¬ 
ity? 
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Of  the  two  questions,  the  first  one  is  the  more  crucial,  since  question 
(2)  is  addressed  in  some  detail  in  Section  6.  Here  we  focus  mainly  on  the 
feasibility  of  developing  and  operating  the  OAS. 


(a)  Needs  Map  Database 


The  first  step  in  developing  the  needs  database  has  already  been  taken  by  the 
Agency  in  the  form  of  the  Instant  Architecture  Data  Base  being  developed  by 
ARM  in  Reston.  The  main  sources  for  identifying  both  nodes  and  arcs  of  the 
needs  graph  are  the  functional  and  CINC  architectures,  both  available  from 
within  the  Agency.  Some  CINCs  prefer  to  do  their  plans  on  their  own.  The 
Agency,  however,  is  responsible  for  reviewing  all  C3  systems  requirements 
and  master  plans  which  can  provide  the  Agency  with  access  to  information 
necessary  to  complete  the  needs  map. 

The  major  missing  category  of  information  is  the  importance  of  each 
link  (Section  5,  (a)l).  As  this  is  determined  mainly  by  the  consequences  of  the 
need  not  being  satisfied,  the  information  should  be  obtained  by  expert  ratings 
(e.g.,  Delphi  Method).  This  involves  opinions  from  without  the  Agency  and 
will  require  establishing  a  formal  non-voluntary  procedure,  (one  possibility 
is  the  IIP). 

The  construction  of  the  needs  map  database  is  a  major  undertaking. 
Nevertheless,  the  efforts  involved  are  not  an  order  of  magnitude  greater  than 
those  required  for  developing  the  Instant  Architecture  Data  Base.  In  our  pre¬ 
liminary  estimate,  if  only  joint  tactical  operations  are  considered,  it  is  pos¬ 
sible  to  complete  the  development  and  testing  of  the  needs  database  within 
an  18  months  time  frame. 
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(b)  The  Projects  Database 


This  is  actually  an  extension  of  the  JIMS  system  which  is  already  opera¬ 
tive.  All  the  necessary  information,  in  addition  to  the  JIMS  information, 
is  available  from  within  the  Agency.  Information  on  status  of  completed 
projects  (Section  5,  (b)6)  may  require  the  institution  of  standard  procedures 
for  feedback  information  as  defined  and  described  in  Section  1,  (e).  A  de¬ 
cision  must  be  made  as  to  how  frequently  should  the  status  of  each  type 
of  projects  be  updated.  Should  it  be  updated  periodically  or  at  completion 
points  of  pre-determined  phases  of  the  project.  Estimated  probabilities  of 
completion  times  and  degrees  of  implementation  may  be  obtained  by  expert 
judgments  combined  with  the  historical  data  on  completed  projects  stored  in 
the  database.  The  degree  of  detail  of  the  information  and  accuracy  of  esti¬ 
mates  is  to  be  determined  by  the  desired  types  of  indices  (measures)  required 
from  the  OAS. 

The  construction  of  the  projects  database  and  its  maintenance  is,  in 
our  opinion,  well  within  the  capability  of  the  Agency.  The  completion  time  of 
such  a  project  is  estimated  as  being  significantly  shorter  than  the  completion 
time  of  the  needs  map  database. 


(c)  Procedures 


Most  of  the  needed  procedures  (software  programs)  are  included  in  the  stan¬ 
dard  database  package  to  be  used  by  the  Agency.  The  special  additional 
programs,  to  be  determined  by  the  features  required  of  the  OAS,  are  rather 
straightforward  and  do  not  require  high  mathematical  skill  or  understanding 
of  elaborate  algorithms.  The  Agency  does  have  the  talents  needed  for  either 
developing  the  required  software  in-house  or  supervising  its  development  by 
am  outside  contractor. 


In  conclusion  we  are  quite  convinced  that  a  development  of  an  OAS 
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along  the  lines  suggested  in  this  study  is  well  within  the  capability  of  the 
Agency.  The  work  can  be  done  in-house  completely  or  partially.  This  de¬ 
pends  less  on  the  availability  of  in-house  talents  and  more  on  the  priority 
given  to  such  a  project  and  the  urgency  of  its  completion.  Use  of  outside 
contractors  may  significantly  shorten  the  completion  time  of  the  system. 

As  for  question  (2),  examples  of  possible  meaningful  measures  of  overall 
effectiveness  and  interoperability  are  outlined  in  Section  6.  The  issue  here, 
in  our  opinion,  is  not  the  question  of  feasibility,  it  is  the  question  of  which 
measures  are  desired  versus  the  cost  involved  in  collecting,  storing,  managing 
and  updating  the  necessary  information  for  producing  these  measures.  In 
general,  the  lower  level  of  aggregation  measures  (indices)  require  finer,  more 
detailed,  breakdown  of  the  data.  This  will  be  reflected  in  the  development 
and  management  cost  of  the  OAS. 


(d)  Implementation 


Assuming  the  Agency  decides  to  develop  an  OAS  of  the  type  suggested  in 
this  report  we  recommend  that  the  following  next  steps  be  taken: 


(1)  An  in  depth  study  for  determining  the  measures  (indices)  to  be  produced 

by  the  OAS,  bringing  into  consideration  the  need  and  usefulness  of  each 
measure,  transparency  of  its  meaning  to  the  Agency  and  other  possible 
users,  ease  of  producing  it,  additional  data  (cost)  required  for  producing 
it,  resulting  byproducts  and  its  inter-relations  to  other  measures.  This 
study  should  also  define  the  data  to  be  maintained  in  the  two  databases. 

(2)  Development  of  criteria  and  procedures  for  ranking  importance  of  links 

and  for  determining  users  requirements  satisfaction,  as  defined  in  Sec¬ 
tion  5.  This  step  should  be  taken  simultaneously  with  step  1. 

(3)  A  quantitative  study  for  estimating  (i)  the  amount  of  data  collection  and 

processing  required  for  initiating  and  maintaining  the  system,  (ii)  the 
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one  time  initiation  investment  costs  and  (iii)  the  ongoing  maintenance 
costs. 

(4)  Design  of  the  OAS  (databases,  computer  programs  and  procedures  for 

collecting,  managing  and  updating  the  data). 

(5)  Development  of  the  databases  and  the  computer  programs  and  collection 

of  the  required  data. 

(6)  Testing. 

(7)  Putting  the  OAs  into  use. 


These  steps  should  not  delay  the  continuation  of  the  development  of  the 
JIMS  and  Instant  Architecture  databases  with  a  view  of  integrating  them  in 
the  design  of  the  OAS.  This  will  require  coordination  of  the  teams  involved 
in  the  three  projects  (JIMS,  IADB,  OAS)  since  the  OAS  will  incorporate 
the  JIMS  and  IADB  and  at  the  same  time  will  enhance  their  usefulness  as 
operational  tools. 

The  option  of  stopping  the  development  of  the  OAS  should  be  main¬ 
tained,  at  least  through  the  completion  of  step  (3).  Should  the  amount  of 
required  data  and  resulting  costs  turn  out  to  be  prohibitive  or  should  deter¬ 
mining  the  measures  of  interests  be  found  an  elusive  task  (which  we  do  not 
expect  to  be  the  case),  the  option  to  abort  before  committing  the  bulk  of  the 
resources  must  be  open. 


(e)  Alternative  Systems 


It  is  possible  to  develop  more  modest  and  restricted  assessment  systems  which 
will  require  considerably  less  resources  in  terms  of  both  investment  and  main¬ 
tenance  costs.  The  most  viable  such  system  will  be  based  on  the  bottom- up 
approach  as  outlined  in  Section  4.  This  report  does  not  elaborate  on  the 
outlining  of  such  an  alternative  system.  At  a  meeting  with  the  Director 
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and  a  group  of  high  echelon  Agency  personnel,  which  took  place  on  August 
10,  1990,  it  became  quite  clear  that,  if  feasible,  a  top-down  based  OAS  is 
preferred. 


